{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7.2.1回归RVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目标变量t的条件概率分布为:\n",
    "$$p(t|x,w\\beta)=\\mathcal N(t|y(x),\\beta^-1)$$\n",
    "均值有线性模型给出:$$y(x)=w^T\\phi(x)$$\n",
    "写成SVM相类似的形式:\n",
    "$$y(x)=\\sum_{n=1}^Nw_nk(x,x_n)+b$$\n",
    "参数的数量为$M = N + 1$,$y(x)$与SVM的预测模型具有相同的形式,与SVM的情形相反，没有正定核的限制，基函数也没有被训练数据点的数量或位置所限制。\n",
    "\n",
    "似然函数为:\n",
    "$$p(t|X,w,\\beta)=\\prod^N_{n=1}p(t_n|x_n,w,\\beta)$$\n",
    "引入w的先验分布，为每一个$w_i$,都引入一个超参数$\\alpha_i$:\n",
    "$$p(w|\\alpha)=\\prod_{i=1}^N\\mathcal N(w_i|0,\\alpha_i^{-1})$$\n",
    "当我们关于这些超参数\n",
    "最⼤化模型证据时，⼤部分都趋于⽆穷，对应的权参数的后验概率分布集中在零附近,⽣成了⼀个稀疏的模型。\n",
    "w的后验分布为:$$p(w|t,X,\\alpha,\\beta)=\\mathcal N(w|m,\\Sigma)$$\n",
    "均值为:$$m=\\beta\\Sigma\\Phi^Tt$$\n",
    "方差为:$$\\Sigma=(A+\\beta\\Phi^T\\Phi)^{-1}$$\n",
    "\n",
    "$\\Phi$是NxM的设计矩阵，\n",
    "元素为$\\Phi_{ni}=\\phi_i(x_n)(i=1,...,N),\\Phi_{nM}=1(n=1,...,N),A=diag(\\alpha_{i})$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "边缘似然函数为:\n",
    "$$p(t|X,\\alpha,\\beta)=\\int p(t|X,w,\\beta)p(w|\\alpha)dw$$\n",
    "对数边缘似然函数为:\n",
    "$$\\begin{align}lnp(t|X,\\alpha,\\beta)&=ln\\mathcal N(t|0,C)\\\\\n",
    "&=-\\frac{1}{2}\\{Nln(2\\pi)+ln|c|+t^TC^{-1}t\\}\\end{align}$$\n",
    "其中C为NXN矩阵:\n",
    "$$C=\\beta^{-1}I+\\Phi A^{-1}\\Phi^T$$\n",
    "\n",
    "关于超参数$\\alpha$,$\\beta$最大化边缘似然函数得到$\\alpha^*,\\beta^*$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "预测分布为:\n",
    "$$\\begin{align}p(t|x,X,t,\\alpha^*,\\beta^*)&=\\int p(t|x,w,beta*)p(w|X,t,\\alpha^*,\\beta^*)dw\\\\&=\\mathcal N(t|m^T\\phi(x),\\sigma^2(x))\\end{align}$$\n",
    "\n",
    "w为后验均值m,方差为$\\sigma^2(x)=(\\beta^*)^{-1}+\\phi(x)^T\\sigma\\phi(x)$\n",
    "\n",
    "RVM中先关向量的数量⽐SVM中使⽤的⽀持向量的数量少得多。对\n",
    "于⼀⼤类回归任务和分类任务， RVM⽣成的模型通常⽐对应的⽀持向量机⽣成的模型简洁了⼀\n",
    "个数量级"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
